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ABSTRACT 

This report described and analyzed different methods 
for constructing a student's cognitive map in order to study the 
development and use of domain-specific prior knowledge. Cognitive 
maps provide accounts of student trajectories over courses and 
individual or group profiles of learners' strengths and weaknesses at 
the course or multicourse level. Methods are grouped into: (1) 
clinical interview.s with individual students; (2) word-association or 
word-sorting tasks (including free recall); (3) writing methods 
(e.g., writing defii^itions or choosing preferred statements); (4) 
bipolar dimension tasks; and (5) think^aloud methods, such as verbal 
protocols. The free recdl (word association) method is considered 
most appropriate for the planned research. Two figures illustrate the 
concepts discussed. (Contains 32 references.) (SLD) 
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Mediods for mvesUgatiug the stiucture of knowledge 1 
Introduction 



Earlier we stated that there is a renewed interest in schema theories and that prior knowledge plays a central 
role in all of these theories (Dochy, 1992). What a learner already knows provides a frameworic for new 
knowledge and it is the starting point for the construction of a r^res^tation of the cognitive structure. 
An advantage of cognitive structures is that they give the possibility to interpret complex ph^omena like 
information - processing. 

In this article, we will describe and analyze different methods for constructing a stud^t's cognitive map in 
order to study the development and the use of the domainspecific prior knowledge. In literature, the 
con^arison of differ^t cognitive structures or maps is seen as a fundamental method to investigate student 
learning. Such reference maps provide accounts of student trajectories over courses and individual or group 
profiles of learners' strengths and weaknesses at the course or multicourse level (Snow» 1990). 

1* Cognitive Structure 

Being able to represent knowledge is a key of being able to utilize it. Knowledge and skills have been 
represented in educational literature in three kinds of repres^tations: cognitive, cont^t, and curriculum 
structures. 

AusubePs (1968) definition of cognitive structure is: an individual's structure of knowledge (the 
sorts of knowledge people have and how they store it). Rq>resentation is defined as the external 
organization of knowledge in a structure. Cognitive structures allow information to be traced and located 
quickly. A schema of concepts and relationships acts as a unit of memory. 

Representations of cognitive structure have three characteristics (Donald, 1987): 

^ a representation reflects the method of analysis used to produce it, i.e. one must analyze a schema into its 
component parts, relationships, and organizing structure in order to represent it. The kinds of representa- 
tions that exist in the research literature can be categorized according to the methods used to prcKluce them 
and the assumptions governing those methods; 

^ the representations and the methods used to produce them assume differing amounts of structure in the 
universe. This range of assumptions of structure is a prominent dimension in the comparison of disciplines 
(i.e. a disinguish between hard (e.g. sci^tific theory) and soft disciplines on the basis whether a paradigm 
or theoretical structure exists, or whether content and method are idiosyncratic in the discipline). 

the degree to which the representations of cognitive structure are descriptive (i.e. they do not include 
transformations or changes in state) versus goal oriented. 

Cognitive structure is extensive and n-dimensional. Any attempt at description can only hope to 
illuminate part of this whole. The n-dimensionality inevitably leads to a trade-off between the extent and the 
detail of the description. 

"^gnitive structure has two components: the knowledge bits it contains, and how that knowledge is 
organize. Knowledge bits (propositions (= representations in memory of fects or beliefs), algorithms (or 
skills), images (omental pictures), a person's private understanding of the meaning of words) can be quite 
different in size and nature. 

The cognitive structure of experts consists of a knowledge-rq)ertoire (= organisation in the memory from 
problem-situations, declarative and procedural knowledge) that is build up by problemschema (= represen- 
tations made for educational puiposes; such a schema should be precise, coherent, and consistent) or it has 
been hierarchical organised (Dochy, 1992). 

Beginners often lack the hierarchical and problem oriented structure. According to Reif (1983) it is 
important to learn these kind of structures. 

Three aspects of the structure of knowledge determine the ability of the student to learn: 

internal integration (= the degree to which different parts of subject matter are interrelated 
with one another and not isolated). 
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connection the degree to which subject matter parts and concepts are connected with 
the prior knowledge) 

* correspond^ce (= the degree to >^ch students* cognitive structure corresponds with the 
experts' cognitive structure in a certain domain). 

The proTOsed dimensions of cognitive structure are based on the Gagn6 and White model. This is a static 
model, in which there are four sorts of elements: propositions, intellectual skills, images, and episodes. The 
dimensions are the following: 

1* extent : how much knows a person; 

2. precision of the knowledge : how pi^isely can he/she formulate his/her knowledge; 

3. internal consistency : are all parts of knowledge compatible; 

4. accord with r^litv or generally accepted truth : are there differences betwe^ knowledge 
and leality; 

5. variety of types of element : is there imbalance between the verbal knowledge of a person 
and his/her episodes or skills; the desirable form of cognitive structure is one of a good 
mixture of types of elements; 

6. variety of top ics: one might, e.g., distinguish betwe^ people who are specialized in 
knowledge of a small number of fields and generalists who have some knowledge of many 
topics; 

7. shape: the form of organization of cognitive structure. If we think of knowledge as a 
network of elements, of whatever types, we can conceive of networks having different 
shq)es and degrees of interlinking; shape could be repres^ted for some purposes by an 
index called association density, the average number of associations per element; 

8. ratio of internal to external associations : the proportion of elements in a chain or net which 
are internal, in the sense of obviously being parts of the subject matter, and the proportion 
which is external, or unessential parts which illustrate the topic (rather than to form a vital 
part of it); 

9. availability (of knowledge) : two people may know the same things, but differ in the ease 
with which they recall relevant elements at need. 

These nine constructs, and the practical measures of them (e.g., shape could be represented by an index 
called association density, the average number of associations per element), are not necessarily independent 
of each other. The intent here is to propose a number of constructs which may be useful in describing 
cognitive structure. 

Earlier we stated that we found a general agreement among experts and in literature that cognitive structure 
provides mostly information on the availability of prior knowledge (u^ch can be differentiated into 
constructs 3,4,7 and 8: internal consistency, accord with reality, shape and ratio of internal to external 
associations). 

The most commonly accepted model of cognitive structures in long-term memory is an associative 
network model (e.g., Anderson, 1976, 1983, in: Martin, 1985)). A netwoik consists of a set of nodes or 
points interconnected by a set of lines or arcs. Within this model, the nodes of the network represent 
concepts, and the lines are links among the concq)ts (Dochy and Bouwens, 1990). The links may vary in 
strength, resulting in some concepts being more strongly associated than others. Long-term memory is 
hypothesized to consist of an enormous number of such networks or stiuctures that themselves are linked 
together in higher-order configurations, perhaps in some sort of hierarchical manner. 
Figure 1 represents a portion of the cognitive structure of a medical student that centres around the concept 
of hart-attack. 

Information is retrieved from an associative network through spreading activation. Specific nodes of the 
network are activated, and the activation then spreads through linked nodes. 
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oau causAio roiatio 
lilen Identitoltsroiafto 



Figure 1: Example of a network structure: a part of a medical students' knowledge 
concerning *hart-attack* (Adapted from Boshuizen, 1989). 



2. Methods for Recording Cognitive Structure 

Several techniques/methods have be^ used in attenq)ts to probe the learner's structure of knowledge. We 
group these methods into S categories: 

1«) Clinical interviews with individual students; 
2«) Word-association or word-sorting tasks; 

3«) Writing methods, i.e. asking learners to write definitions, or to choose a 

preferred statement from several correct ones; 
4«) Bipolar dimension tasks, i.e. tasks which involve bipolar dimensions on which 

an idea is rated; 
5«) Think aloud methods, i.e. verbal reports (protocols). 

These categories of techniques will be discussed in the following paragraphs. In describing these, we will 
try as far as possible to take account of the criteria which should be specified for any valid method for 
assessing cognitive structure (Shavelson, 1974; in: Jonassen, 1987): the stimuli used to examine cognitive 
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Structure; the instructions to subjects; the context in which the measurement takes place; the nature of the 
subject's response. 



2A. Clinicel Interviews 

These techniques were originally used in clinical psychology. 

Piaget (1929) described the ess^ce of the method as follows: *'the clinical examination is dependent on 
direct observations, the practitioner lets himself be led, and takes account of the whole of the mental 
context". 

The clinical interview method, as originally used by Piaget, takes two formes: verbal and revised. 
The verbal clinical method involves flexible questioning of individual children on a totally verbal level; 
concrete objects do not serve to illustrate or represent the problem posed by the interview. 

Exanq>le provided by Gelman (1980, p. 65; in: Ginsburg) 

7; What 's the biggest number you can think of? 
S: A billion, 

I: Is that the biggest number? 

S: No. 

I: What is? 

S: I don *t know. 

I: Why don V you know? 

S: Numbers never end. 

I: What if somebody told you that a googol is the biggest number? 
S: rd say that numbers never end. 
I: Could you prove it? 
S: No. 

I: What if you add 1 to a googol? 
S: A googol and I. 

In the revised clinical method, the data of interest are both veri>alizations and aspects of nonverbal 
behaviour. Piaget attempted to improve the verbal method by using concrete objects to illustrate the problem 
to be solved. 

The classical example is the observation of numbers, as described by Piaget and Szeminska (1952): 

Hab (5:3) began by putting 9 sweets opposite the 6 of the model, but made the row the same lengtfi. Ihat^s it —Are they 
the same?''rm not sure. —Where are there nu)re?'-Ihere throw of 9, close togethery-What must we do then?— (She put 
6 opposite 6 of the model and removed the resty-Ihe 6 of the model were then closed upy-Are they the same ?''No. -Are 
there as many here (htodel) as there?— No, there (copy) there are more— Is there more to eat on one side than on the 
other, or are both the same?— I shall have more to eat.— Make them both the same, then —(She removed 2. then made 
the one-one correspondence, and finally put the two back when she found they were missing!) [p. 79] 

Piaget developed the clinical method in order to achieve three aims central to the study of cognitive 
development: 

1. the elicitation of intellectual activities; 

2. the specification of the nature and organization of cognitive processes; 

3. the evaluation of the child^s level of cognitive competence. 

Ad.l.) The clinical method attempts to give children the opportunity to engage in various intellectual 
activities, like the speculation on the origins of the sun. 

Ad.2.) Once relevant intellectual activities have been elicited, the cognitive processes underlying them need 
to be examined in detail and described as precisely as possible. Piaget usually describes these processes in 
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terms of the "structures** of intelligeace, general mental capacities presumably underlying performance on a 
wide range of tasks. 

Note that elicitation of intellectual activities and specification of cognitive processes operate at several levels 
of specificity. The begiiming stages of research may be said to involve discovery : our knowledge of 
cognition is so limited that it is necessary to begin most research investigations with the aim of discovering 
relevant phenon^na and processes. Next the investigator wishes to go beyond the discovery stage to obtain 
further empirical data of interest and to develop precise descriptions of cognitive processes: specification . 
Furthermore, discovery and specification app^ to be cyclical processes (discovery specification 
discovery specification -*» etc.). 

Ad.3.) Piaget's theory aims at establishing underlying compet^ice -the child^s highest ability at this current 
stage of development- and does not oft^ deal with typical performiance -the child^s ordinary beht^viout on a 
particular occasion. 

The investigation of competence focuses on three areas: subjective equivalence (the child 
understands the question in the way it is intended), seriousness of response (the child takes the task serious- 
ly), and strength of belief (the child*s response is deeply held). 

A related and more recent iqiproach or method, until now based on interviews, was developed in Molndal, 
Sweden at Gothenburg University. This 'phenomenographic* approach offers techniques for systematic 
interviewing or recording of student structuring. This phenomenography of learning, still under 
development, is an approach to describing learning as a change betwe^ qualitatively different concq)tions 
based on students* cognitive structures (Johansson, Marton &Sv©nsson, 1985; Marton, 1987; Marton, 1988; 
Kroksmark, 1990). Although int^isive interviewing may often be uneconomical, some of these procedures 
can be used selectively, some may be computerized in future (Snow, 1990). 

2.2. Word-assodation taste 

A method for assessing cognitive structure is to generate a map of it, i.e., a coignitive map (Jonassen, 
1987). The techniques available for mapping cognitive structure are all based upon the concq)t of semantic 
proximity of words or constructs. That is, the more proximate words are to each other in an individual's 
cognitive structure, the more semantically related they are. This notion of proximity assumes that we can 
rq^resent knowledge in terms of geometric space, that is, map knowledge structures. 
Cognitive maps are produced from behavioral measurements and interpreted as abstract representations of 
one*s laiowledge of some subject matter structure (Shavelson, 1985). It is based upon an associative network 
theory of long term memory. 

Preece (1976) compared the three most prominent methods for mapping cognitive structure: 

(1) free word association/ free recall , i.e. a paired association task using the same stimulus 
word repeatedly; 

(2) controlled word association : this requires the learner to associate words in rank of order of 
importance. Preece generated relatedness coefficients (measures of overly between two 
word association hierarchies) for each pair of words in the same content domain; 

(3) tree construction tests ; this requires learners to construct a linear graph (tree) in which the 
vertices (intersections) are the words and the connecting lines the relationships. The two 
most closely related key concepts, according to the learner, are linked with a line 
numbered one. Then the key concept most closely linked to either of these was chosen and 
linked with a second line. The learner proceeded imtil all key concepts had been placed in 
relation to the others. The resulting tree structure displayed the closest relationships 
between key concepts in order. 

With the word-association iUEree^!resi>onse method (Shavelson, 1974), the subject is shown a 
stimulus word and asked to respond with the first word or all the words he can think of. The raw data, 
then, are the subject^s list of responses to each stimulus word in the test. 

Example provided by Shavelson (ref. 14, p.41): 
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This is a test to see how many words you can titink of and write down in a short time. 

You wiil be given a key word which represents a concept in "physics", and you are to write down as many other words 
as you can which the key word brings to mind as you can. You wiU probably not be able to fill in all of the spaces on a 
page, but do the best you can. Be sure to think of the key word after each word you write down, because the test is to 

see how many other words the key word makes you think of 

For example, suppose I asked an electrician to write down as many words relating to electricity (his speciality) as he 
could think of when given the word "Conductor", 



He might put down the following: 

Conductor 

Conductor metal 
Conductor resistor 
Conductor charge 
Conductor 



Conductor transmit 
Conductor 



You will notice that as an electrician, he did not put "train" or "cable car" or "guide", since they do not relate to the 
electrical concept of "conductor". In this same way, you should think like a physicist when you respond to the concepts. 
You will have one minute on each page. 1 will tell you when to go to the next page. 

From these data, a number of differ^t scores are commonly derived: 

a. the total number of responses to a stimulus word; 

b. the average number of responses to the stimulus words; 

c. the number of responses of a particular kind in the responds list (e.g., only responds words that 
constitute concepts in an equation defining the stimulus word); 

d. the overlap of responses lists for pairs of concepts* 

The number, kind,, and ordfsr of responses in the list, and the overlap of response lists, take on particular 
importance in interpreting w:orc! association data. They ars used to make inferences about cogmtive structure 
with the acknowledgement that the structure is not necessarily represented in its complete form or without 
some distortion. 

Two disadvantages of this method are (Hirschman,1982) that: 

L) the data provide no information on the nature of the association, and in particular, on the 
positional relationship of an evoked response to the stimulus; 

2.) the stimulus list has to be developed a priori and assumed to form a specific hierarchy* 



In the ^ph construction method (Shavelson, 1974), respondents are given an alphabetical list of 
key terms (sometimes derived from word association methods) and asked to build a linear tree graph by 
connecting pairs of words (nodes on the graph) with a line representing the "similarity** between all pairs of 
words (nodes) on the gmph {-^ they are asked to arrange these terms in a pattern that represents illation- 
ships among them, with nodes as concepts and lines as relationships). The mw data, then, are the distances 
betwe^ all pairs of words (nodes) on the gtdph. From these "proximity" data, various scaling techniques 
can be used to examine the structure underlying them. With this method, the order in which words are 
connected by lines on the graph and the resultant proximity among words take on particular importance in 
examining cognitive structure. 

Example 

On the top of the next page is an alphabetical list of N terms. Read the list carefiilfy several times. From the list of N 
terms, pick the two terms which you think are "tnost similar" to each other. Write the pair you have chosen in the 
middle of the page and connect them with a line. Label the connecting line 1, 
Now you have two options as follow. 

Option 1: You may go carefully over the remaining terms in the list (which now includes N'2 terms) and pick 

the term which you think is most similar to either of the two terms you have already selected. Write 
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this term down on your paper and connect it tot the appropriate term already selected. Label the 
connecting line 2, 

Option 2: You may look over the remaining terms on the list and dec?Je that two of them are more similar to 

each other than any one of them is to either of the two terms already selected and joined together. If 
so, you may select tSiese terms and write them down on the paper, Just as you did with the first pair. 
Connect the two new with a line and label it 2* 

After taking option J or 2, proceed in exactly the same way. Search carefully tlie remaining terms on the list and 

continue with option J or 2. Wten you take option 2 you add a term to an already linked group of terms (which w 

caUed a tree). Wien you take option 2 you start as a new tree. 

As you proceed, you have a ^d option: 

Option 3: If you find that you have made several trees, you may want to comxect any two of them together. If 

you find two terms, on two separate trees, that are more sanilar to each other than any other term 
on the remaining Ust is to any ot^ier term on the trees, you should connect these terms (and Htvu. 
connect the two trees). Label the connecting line according to the sequence already started. 

In short you have the following options: 

1. Adding a mw term to one of the trees you have already made. 

2. Starting a new tree with two new terms. 

3. Connecting two separate trees. 

Please continue in this way until all N terms have been exhausted and until you have connected all separate trees into 
one tree. Remember, write down each term just once. You may find it helpful to cross out the terms in the list as you 
write them down). When you have finished, you will have made one tree of N terms cotmected by N-l numbered lines 
(adapted from Fillenbaum and Rapport), (ref.656, pp 15-26). 

Sorting techniq ues require respondents to sort terms or concept labels (again typically determined 
from word association methods) into various categories on the basis of perceived similarity (relatedness) and 
dissimilarity (e.g., Shavelson» 1974). 

In the card sorting method (Shavelson & Stanton, 1975) subjects receive cards with domainspecific concepts 
printed. Each card contains one concq)t. Participants are requested to sort the cards on as many stacks as 
they choose. Cards in each stack should have a high degree of simUarity. 

For each participant a matrix of similarity is to be constructed. Each combination of concepts gets a certain 
value, i.e. a value of 1 id both concepts are in the same stack, a value of 0 if they are not. Subsequently, 
the matrices of all subjects are summarized into one matrix. The maximum value of each cell will be equal 
to the amount of participants, the minimum value then is zero. 

The concept structuring analysis technique fConSA^T) was devised by Chanqiagne and Klopfer 
(198i) as a way of eliciting graphic rq)res^tations of stud^ts' cognitive structure. The basic procedure of 
this method is not the same as tiie above described method. For stud^t interviews using the ConSAT, the 
concepts were written on small cards, and the student was asked to id^tify vAnch ones he/she recognised, 
and then to provide a definition for the recognized concepts. Finally, the student was asked to arrange the 
recognized concq>t3 on a sheet of paper **in a way that repres^ts how you think about the words and about 
the relationship betwe^ them**. When the arrangem^t was convicted, the grouped concepts were circled 
and the student was asked why he/she made this arrangement and to explain particular groupings. The 
student* s answers were tape-recorded and written on the sheet containing the arrangem^t. The resulting 
cognitive structures were subsequently divided into categories of structures based on several criteria. These 
categories are hierarchical organised in which the top category is best integrated, systematically organised, 
etc. 

The method used by Champagne et al. shows the disadvantage that the classification of structures involves a 
reasonable reduction of richness of the data. Relations between different concepts for exanq)le cannot be 
traced back. 



2.3* Writing methods 

Schaefer (Sutton, 1980) includes writing a definition as a technique in his investigations of the concepts of 
growth. Because the respondent has to compose a statement in his own terms, he has the opportunity to 
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consider all the ideas available to him about the topic. No doubt the act of composition can help to make 
more finn any incipient of the knowledge which is not yet well established. 

Kemper and Hodgson (Sutton, 1980) used the technique of allowing the respondent to choose one 
definition firom several all of which can be regarded as correct. This has the advantage that it is accessible 
to pupils v/ho may find composing their own definition difficult, and therefore give up; and it also lends 
itself to quantitative processing of the results of the investigation. 

Analysis of a writing protocol; Hayes and Flower (in: Gregg and Steinberg, 1980) describe a model 
of the writing process derived through protocol analysis (-> a protocol is a description of the activities, 
ordered in time, which a subject engages in vjinle performing a task). 
The unique features of the model are: 

1. ) it identifies not only subprocesses of the conq>osing process, but also organization of those 
subprocesses; 

2. ) minor variations in its simple control structure allow it to describe individual differ^ces in 
conq>osing styles. 

To facilitate the analysis of writing, they have divided the writer*s world in three major parts (figure 2): 

* the task environment, -» includes everything outside the writer*s skin that inferences the 
performance of the task (e.g., information relevant to the writer*s motivation); 

* the writer*s long term memoiy, -s» vmters have knowledge about many topics (e.g., auto 
mechanics) and audiences (e.g., children); 

* the writing process, writing consists of three major processes: 

- plaiming (-> junction: to take information from the task environment and from t1.e long 
term memory, to use it to set goals and to establish the writing plan to guide the producti- 
on of a te^t that will meet these goals); 

- translating (-> Auction: to take material firom memory under guidance of the writing plau 
and to transform it into acceptable written English sentences); 

•reviewing (-* fimction: to improve the quality of the written 
text). 
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Figure 2: Structure of the writing model (Gregg & Steinberg, 1980): 
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2A. Bipolar ^dimension tasks 

An altogether different t^proach to the investigation of personal meanings is to ask pupils to rate a 
concept in terms of its position on some sort of continuum (based on the essence of the Osgoods technique, 
1957). A stimulus word (say ^bacteria*) may be seen as in a certain position on scales of bad-to-good» weak- 
to-powerful, pa^sive-to-active. The results can be displayed in a two-dimensional or three-dimensional 
gfBph. The technique is of course concerned exclusively with personal and private meanings, not public 
definitions, and with these particular scales the technique may reveal the emotional connotatioiis of the 
stimulus word rather than its logical connections. Words of very different denotative meaxiing can ^pear in 
similar positions in the 'semantic space*. 

Whereas Osgood's test offers ready-made bipolar scales or ^dimensions* on which to rate a giv^ 
stimulus word, one can also try to discern what dim^ssions are spontaneously being used by the learner , 
using Kelly's 'repertory grid* technique (described by Fransella and Bannister, 1977). A child may be given 
three objects, or asked to select three at random from a large number that are offered. He is th^ asked to 
consider any two of them which resemble each other in some way and are differ^t from the third, and to 
say how they are different. Repetition of this procedure creates a list of bipolar dimensions which form the 
criteria which the child is consciously or unconsciously using in making sense of the infonmtion around 
him. In a sense we are here identifying the kind of *nmtal spectacles* with v^ch he views the world. 
Although Kelly's technique was designed to elicit continuously variable 'constructs* (i.e. bipolar continua) 
rather than classification concepts, in practice the respondent may make use of such concepts in doing the 
task. For exan^le, a child who selects from some optical instruments a periscope, binoculars and a camera, 
and separates the former 'because it worics on reflection only, while the other two use refraction*, clearly 
has the concepts of reflection and refraction and can use them as mental 'tools'. 

2.5. ^'Think aloud" methods 

In recent years some researchers (e.g., Ericsson & Simon, Ginsburg, Black) have defended 
protocol analyses as a data base for exploring tasks (e.g., prose comprehension, problem solving, 
mathematical reasoning, writing) and mechamsms involving different knowledge domains. 
Graesser (1981) contrasted complex verbal protocols with simple response measures as methods for 
investigating comprehension mechanisms. 

Complex verbal protocols include "think aloud" protocols, Q/A protocols, question asking protocols, recall 
protocols, and other verbal protocols. Q)mplex protocols are qualitative and supply rich patterns of 
information. 

Simple response measures include reading times, priming latencies, lexical decision times, the duration of 
eye fixations, ratings of stimulus items, and other measures which normally have a quantitative scale. 

In a verbal, or "think aloud* pmtocnr subjects are asked to say aloud everything they think and 
everything that occurs to them while performing the task, no matter how trivial it may seem. These think 
aloud data are re;x)rted in a protocol which provides the raw data for protocol-analysis. 
Thinking aloud is often used in research into problem solving* Subjects are asked then to solve a problem or 
the primary task and to talk aloud meanwhile (the secondary task). An intervention is only made when the 
subject fails to perform one of the tasks. 

A great advantage of this method is the immediate availability of qualitative data. Disadvantages are the 
complexity and the possibility for subjective interpretations of the data* 

Thinking aloud in the model of Ericsson en Simon (1980), is a process during which all elements 
of STM (i*e* information that is *in focus*) are translated into natural language (spoken by the subject). 
Subjects mostly do not verbalize all information in the STM. Therefore, the protocol will often contain only 
a part of the elem^ts. 

From the Ericsson and Simon model, factors can be derived that influence 'thinking-aloud': 
a. the ability to verbalize elements of STM; 
. b. the nature of the primary task; 
c. the verbal capacities of the subjects; 





Methods for investtgatiog the stnicture of knowledge 10 



d. 8 person^s capacities related to the primaiy task. 

The talkiog aloud procedure (Ginsburg, 1986) permits investigation of the complex activities that 
constitute problem solving and the exploration of the internal symbolic mechanisms that underlie it. Since 
the investigator plays a limited role in the proU)col collection, the data can be viewed as relatively unbiased 
accounts of the individual subject's problem-wlving performance and may shed light on individual 
differ^ces in cognitive processes. 

Consider a condensed version of a talking aloud protocol involving cryptarithmetic, adapted from Niwell & 
Simon (1972, p.329). 



The subject is presented with the problem DONALD + GERALD = ROBERT, and, given that D is 5, is asked to 

determine the numerical values of each letter. 

I: I will give you that D is 5 in this problem. Please talk, 

S: Well D,..giving D^5 automatically makes Ta zero. Could you make Ta zero? 
I: Tis a zero, 

S: Because 5 plus 5 is equal to 10, And that's simple form the problem. And looking at the leftmost colunm, you can see 
that R is either 1 or 2 greater than O, but that doesn 7 seem to help very much at this point. In the second column 
having the two L 's equal, and also the two A 's equal in the third column^ doesn V seem to help much at this point ei- 
ther,,,,. 

The aims of investigators, who employ the talking aloud method, are to elicit and describe the integrated 
activities constituting complex problem solving in intelligent adults and to identify the cognitive processes 
and internal symbolic mechanisms underlying adult problem solving. It is argued ttmt both of these aims are 
best achieved by obtaining data from a situation in which the subject deals with a challenging problem and 
in which there is a minimum of investigator intervention. 



Different types of TOL (Thinking-Out-Loud) tasks (Kieras and Just, 1984) for collecting data can be 
distinguish^: 

^ Sentence-by-sentence talking: the subject is asked to talk after each sentence in the text. The talking 
continues imtil the text is 
completed; 

Selective talking: subjects have to talk at only particular points in the text; 
*** After the fact talking: subjects have to talk about their cognitive experiences after the feet (-> risky, 
because memory is too fallible to allow for accurate reporting of earlier mental stat^). 

One's specific theoretical and empirical goals will dictate which variant of the TOL task will be most useful 
in collecting the TOL data. 

When thinking about TOL data there are several points about their use to keep in mind (Ericsson 
and Simon, 1980): in g^eral, limits on what is available to be reported upon, what can be remembered, 
and on the human ability to offer explanations or justifications for one's own behaviour should be respected. 
Furthermore, TOL data provide a sample of what's on the subject's mind during the task. 

A particular elaboration of the TOI method is the teach-back procedure. Following a sequence of classfoom 
or textbook learning of some hours, the student is asked to teach the same material to anpth<dr person with 
no prior knowledge ( Snow, 1990). The session is recorded and then segmented into chunks using Chi's 
(1978) interresponse latency rule. These chimks are analyzed to yield measures of the number of units per 
chunk, correctness and completeness, degree of verbatim 

recall versus degree of elaboration, personalization, reorganisation and the amount of common or 
idiosyncratic misconceptions. Research by Gray (in Snow, 1990) showed that the results of this teach-back 
method correlated strongly with acliievement tests administered before and after instruction. Moreover, the 
method provides the possibility to ndentify qualitatively different forms of knowledge structuring. 
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Analysis of TOL protocols can be done on the basis of the method proposed by Breuker (1982), called 
"analysis-by-synUiesis". This method consists of a continued r^wdring and specifying of a theory. The 
protocols need to be interpreted. A first step in Breukers* method is the design of a interpretation-theory: a 
description bow to interpret information in a protocol. For this purpose, taskanaiysis and a sample of 
protocols can be used. 

A first important decision is the choice of categories to distinguish and the relations between the categories. 
Further, this theory must be specified in a coding system for statements in the protocol. These codes are the 
oper>iionalisations of the more abstract categories. 

In a first phase of the analysis, a tiy out of these codes on a selection of protocols will he nece^ry. This 
can result in some changes in the coding system or the chos^ categories. The complete theory will be 
developed by means of certain amount of these *cycles% until the analyzing system fits to the format of 
protocols. Then, definite coding of all protocols can take place. 

These will provide a basis for evaluation of the theory related to the individual differences between subjects 
and the generalization towards new protocols. 

Ericsson and Simon (1984) also suggest for the analysis of protocols to split in units and to code them in 
order to avoid influence of the context and the preceding or proceeding processes. 

3. Discussion: Applicability of the Knowledge Base Structure Methods for Research and Educational 
Practice 

As advocated earlier, a method for investigating the cognitive structure of stud^its could provide useful 
information for our research into the prior knowledge base of stud^ts. Especially the accessibility and 
relevance of the prior knowledge at a certain moment can be mapped. The purpose of this article is to seek 
for a priority in different methods related to this goal. 

In our point of view there are several advantages and disadvantages that should be taken into account when 
considering these methods in investigating prior knowledge of students. 

First, in a research context, a method providing cognitive structures gives information on the availability 
and relevance, the accessibility of prior knowledge and the information processing (Shavelson, 1985; 
Martin, 1985). This information could be vital for seeking an explanation for the different effects of prior 
knowledge (Hirschman, 1982; Lampert, 1986). 

Secondly, apart from providing unique information, these methods are, applicable to almost any subject- 
matter. 

On the other hand, these methods are only applicable for a small group of students and thus not in 
educational practice with large groups of students (Snow, 1990). This is mainly due to the large amount of 
work required for gathering and analyzing data. 

Generally speaking, data from measures of cognitive structures have been analyzed in three ways (Martin, 
1985). Qualitative analyses have depended on detailed, descriptive accounts of the various dimensions of 
cognitive structure. When such data are quantified , they are analyzed with commonly employed descriptive 
and inferential statistics or subjected to complex scaling methods that atten^t to provide sophisticated 
statistical solutions to the description of dimensions or factors that define various cognitive structures. The 
choice of analytic methods employed depends, of course, on the exact purposes of individual researchers. 

As measures of structure, Shavelson and his associates have applied several psychometric techniques that 
malce use of coefficients of concept interrelatedness. Word association, graphic coustruction, and/or card- 
sorting tasks have been used to assess the similarity of key instructional concepts. Then, multidimensional 
scaling, hierarchical clustering, and graphic representation of these judgements are used to reflect students' 
cognitive structure of those concepts. 

Multidimensional scaling uses proximal scaling techniques to place concepts in relation to each other on a 
matrix (Nagy, 1978). This method assumes that relationships between concepts are unidimensional rather 
than multifaceted or logical in nature. The method does, however, show the degree of relationship among 
the set of concepts. 
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Methods of analyzing course data using proKimal and logical methods in conjunction could be expected to 
more accurately dq)ict course content. One of these methods is propositional analvsifi . in vAdch tfie principal 
propositions in a course are extracted from text or lecture material and then subjected to tests from subject 
matter experts and students to determine the underlying structure in a coivse (Donald and Nagy, 1985). 

For the purpose of research into the structure of an individual's knowledge base, it is necessary that a 
method is used which holds possibilities for indivi(I:!al associations or thoughts and free response 
(Hirschman & Wall^dorf, 1982; Ericsson & SicK>n, 1980). Checking the differ^t methods discussed 
above, only the free recall method and think aloud method could be useful. A more time consuming method 
could be to use the concepts resulting from a free recall session as a starting point for a graph construction 
test or a bipolar dimension task. Further, we have to take into account that our goal is to develop 
instruments that are ^)plic&ble in educational practice and useful for a large student population, as are the 
differ^t domain-specific knowledge state tests and knowledge profiles (Dochy, 1992). Therefore, only the 
free recall method seems to be appropriate to our conditions and may add information to our analysis of 
students* knowledge profiles before they follow a particular study traject. 
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